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ABSTRACT 

Evaluation as a political arbitration entity is 
discussed in the case of a multilateral literacy development project 
in the fourth year of operation in a Third world country. An external 
evaluation team was invited to evaluate the project when conflict 
appeared between the funding agency (A) and the technical agency (B) 
over a project-related issue: the extension of an expired 
Instructional Materials Expert (1MB) position. Agency A felt that I ME 
tasks were completed, while B felt that the post extension was B's 
decision. The recommendations of the evaluation team were 
organizational and dealt with personnel issues: (1) an exit da.e to 
transfer project responsibility from A and B to government agents; 
(2) a change from project supervision to government advisor for a 
technical agent in preparation for 0 team exit; and (3) a four-month 
government funded extension of the IMS position. The paper concludes 
that evaluation is political in its genesis, political as a process, 
and political in its effects. This means that evaluators must begin 
with collect ; no political data of relevance to the evaluation action, 
and must learn to take account of political intelligence in the 
process of generating, interpreting, and utilizing technical 
information. (PN) 
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EVALUATION AS ARBITRATION: 
EXTERNAL EVALUATION OF A MULTILATERAL DEVELOPMENT PROJECT 
IN A THIRD WORLD COUNTRY 
By H.S. Bhola 

Evaluation technology confounded by politics. As a 
technology, evaluation seeks to generate needed information about 
contexts, inputs, processes and outputs of development programs 
and projects, systematically, objectively and with exactitude, 
for such information to be used to improve both the efficiency 
and the effectiveness of programs and projects being evaluated (1). 
But as Cohen points out, evaluation is political as well: 
"Evaluation is a technique for mersuring the satisfaction of 
public priorities; to evaluate a social action program is to 
jstablish an informtion system in which the main questions 
involve the allocation of power,- status, and other public goods" 
(2). 

It is now quite well understood that there is more to 
evaluation than doing science. The evaluation process indeed 
entangled with individual politics, organizational politics, and 
the politics of the macro systems within which it is conducted. 
As Carol Weiss has reminded us, programs being evaluated are, 
after all, creatures of political decisions and evaluation 
reports do enter the political arena. Evaluation itself has a 
clear political stance (3). The decision to commission an 
evaluation of a program or a project is a polirjoal decision; and 
the evaluator, in accepting to conduct the evaluation, makes an 



implicit ccxwnitment to the existing social order. The evaluacor 
also accepts the intervention model embedded in the program 
structure; and then makes a political choice as the experimental 
or the participative methodology of evaluation is selected. 
There is a clear relationship between the power structure and the 
choice of indicators of change; and those innocent looking tests 
are ultimately connected with the social mobility systm. 
Politics gets involved in the choice of sites for the evaluation 
exercise; and in the choice of criteria and standards to judge 
the success or the failure of the program. The data from the 
field come wrapped in values; and thus the so-called hard facts 
are soft at the core. Again, politics gets involved, first, in 
the interpretation and, then, in the utilization of results. 
And, finally, there is the politics of "the evaluation of the 
evaluation" itself (4). 

Unfortunately, evaluation theory has yet to catch up with 
the realities of evaluation in the real world. In a recent 
review of evaluation theory, Stufflebeam pointed out that early 
evaluation models all "tended to equate evaluation with the use 
of some preferred techniques" (5). The "participatory 
evaluation" model, mor* than the "advocacy evaluation" model, 
can respond to political realities. Yet, Guba and Lincoln's 
remark that sociopolitical context has been excluded from 
evaluation "as beyond the pale of 'objectivity'" renains largely 
true (6). Cronbach's thesis that "a theory of evaluation must be 
as much a theory of political interaction as it is a theory of 
how to determine facts" (7) throws a conceptual challenge yet to 
be taken up by evaluators in right earnest. 



This paper does not pretend to pick up the theoretical 
challenge and accomplish the theoretical task in one fell swoop. 
But we do hope to take a step towards a political theory of 
evaluation. This theoretical step is taken on the basis of the 
evaluation, recently undertaken by the author, of a multilateral 
development project in the fourth year of its operation and 
located in a least developed Third World country (8). 

The Case: 
Evaluation as Arbitration 

The case presented below is a special case of the politics « 
evaluation. It demonstrates, I think, a special use of evaluation 
evaluation for purposes of arbitration of a dispute between two 
multilateral institutions of technical assistance jointly 
responsible for the same development project. 

Two multilateral technical assistance agencies, UN (A) and 
UN(B), were jointly responsible for a "literacy for development" 
project (Litdev) in a Third World country (Xland) . UN(A) provided 
funding for the project and, through its resident representative 
in Xland, supervised and monitored the project. UN(B) as a 
technical agency was responsible for the professional 
implementation of the project. As such, it selected and 
appointed all professional staff; and, throught then, was 
responsible for all professional and technical decisions as well 
as for resource allocations within the agreed upon budgetary 
limits. The UN(B) did not have a resident representative of 
its own in the country and was indeed represented to the 
Government of Xland through the resident representative of UN (A). 



The arrangement worked quite well, until conflict appeared between 
the two agencies over some project-related issues. It is at this 
time that the agencies involved invited an "external" evaluation 
team to evaluate the project. The team can justly be called 
external because none of the four members of the evaluation team 
was associated with the project as a functionary, nor was any one 
of them currently employed by UN (A) , UN(B) or the Government of 
Xland. 

The evaluation team would be a tripartite team: each of 
the three institutions involved, UN(A), UN(B) and the Government 
of Xland would appoint a member. The fourth member of the team 
would be an outside literacy specialist with experience in 
evaluation in the Third World and would be appointed by UN(E). 
The team would be led oy the member appointed by UN (A) . The team 
would stay in the country for two weeks and follow the procedures 
and methodologies established by mutual agreement bewteen UN (A) 
and UN(B) and accepted by the Government of Xland. 

The terms of reference and the themes of concern 

To anticipate the story somewhat, it was intriguing to 
note that there were two streams of consciousness intermixed in 
the communication behaviors of everyone involved in the evaluation 
project. At one level, they talked about the need for evaluating 
the project. At another level, there was the search for 
arbitration of a dispute. 

This ambivalence was quite clear in the terms of reference 
document provided to the evaluation teat*, while the document 
went to some length in describing the professional needs of the 
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project that the evaluation team had to meet, the document did 
betray the thane of its real concern. For instance, the terms of 
reference dooment began by describing the objectives of the 
evaluation to be to determine how effectively the intended 
purposes of the project were being attained and how effective the 
project was likely to be in the future to contribute to both 
sectoral and national purposes of Xland. Hie evaluation was also 
to identify factors that had facilitated or deterred the 
achievement of intended objectives in the past; and how those 
factors might be manipulated in the future for ever more 
effective actions* This was the language that provided die 
technical rationality for the conduct of the evaluation exercise. 

Hidden under these pretensions of technicality however, 
there was a clear theme of a conflict in search of arbitration. 
The Litdev was a two-man project as far as the foreign technical 
assistance personnel were concerned. One of the posts had 
expired a few months earlier and was now being extended month by 
month from extra-budgetary sources. UN(B) as the technical 
agency responsible for the project wished to extend the post 
because, according to them, sane important tasks still remained 
to be accomplished. However, UN (A) was not convinced. The 
various agencies of the government of Xland came to join the 
issue on different sides, and two opposing coalitions of 
organizations were formed. 

As UN (A) saw it, the tasks of the Instructional Materials 
Expert (IME) were all done. The I ME had been on the Litdev 
project for three years, during which time, curricular decision 
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t had all been made, materials for use in the Litdev had been 

designed and produced. No doubt , there were some additional 
instructional tasks that remained to be achieved , but those had 
been anticipated , at the project planning stage, to be done by 
short-term consultants with specialized technical skills. The 
implication was, of course, that the present IME did not have 
those specializations at the level required. The UN (A) had 
consulted with the team leader of Litdev, a UN(B) appointee, who 
had agreed with their reading of the situation. The local 
authorities took too long to come on the side cf the IME and in 
making the statement that his post did need continuation. In any 
case, the noney for the continuation of the post had run out and 
UN (A) disliked being put in a position where they were forced to 
pay the IME from funds meant for short-term consultants. 
They resented all the cable traffic they had to receive and to 
generate. As far as they were concerned, it was not right to 
personalize an official issue. Seme of the agencies zf the 
Government of Xland were also of the same view. Finally, the 
personal egoes of many of the UN (A) staff were bruised when they 
were characterized by UN(B) as administrators who should not 
be interfering in professional matters. 

UN(B), on the other hand, felt that as the technical agency 
responsible for the implementation of the project, it was for 
them to decide whether the post of the IME needed to be extended 
or not. The UN(B) HQ took their local team leader to task for 
having played in the hands of the local office of the UN (A) and 
for having suggested that the tasks of the current IME had indeed 
been completed; a local conterpart had already been trained; and 
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that the need for short-term specialist consultants was paramount 
to the continuation of the I ME. UN(B) had found money from a 
third party, an international foundation dealing with 
development, but the gesture had backfired. The local treasury 
department of Xland took exception to UN(B) commiting the 
Government of Xland to contractual arrangements with any outside 
institution or government without prior consultation with and 
clearance from the qovernment. 

In the ten months prior to the external evaluation here 
reported, two internal tripartite reviews by UN(A), UN(B), and 
the Government of Xland had been conducted but no agreement had 
emerged. The arbitration role of the present evaluation was 
quite clear to everyone, thought no one talked about it directly 
and forthrightly. 

The two faces of evaluation 

While the vrtiole of the project. team was aware of the 
politics of the evaluation, this author had a special theoretical 
interest in the evaluation (See No. 8 of Notes ^nd References 
below) ; and did in fact develop, as systematically as possible, 
two profiles of the project: (i) a technical profile and (ii) a 
political profile. 

It is not within the scope of this paper to include the 
details of either the technical or the political profile of the 
project. What we need to point out is that a political profile is 
developed by asking political questions in regard to the existinq 
politics at the individual and organizational levels and at the 
level of the social context. As pointed out earlier, in this 

8 



■ I 

particular evaluation, there were no formal instruments that were 
administered. The data were all collected through observation, 
interviews and discussions in individual and group settings. It 
was possible for the author to ask political questions especially 
in individual interviews. Hie fact that this evaluator had 
worked in the country before in other educational and development 
oriented projects helped in establishing rapport and trust. 
Perhaps, this is the best way to go: technical questions may be 
included in formal questionnaires and interview schedules, while 
data for the political profiles are collected more informally. 
The point to be made here is that political data must be 
collected in ernest and not dismissed as so much back-biting and 
mud-slinging deserving of no serious attention. 

Formulation and presentation of recommendations ; 
Concer ns for arbitration met 

The concerns for arbitration were obvious in the way 
recommendations were developed and presented to the various 
stakeholders. At the stage of data collection all parties to the 
dispute had been interviewed and all the various levels of the 
enterprise had been covered from the village to the central office. 

The recanmendations of the evaluation team, quite 
expect^dly, were mostly organizational and dealt with personnel 
issues. It had indeed been made quite clear in personal 
briefings, by both UN (A) and UN(B), that they wanted to have a 
clear resolution of the IME issue and wanted the recommendations 
of the evaluation team to be clear, unequivocal and such that 
could be acted upon both professionally and administratively. 
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Technical recommendations made by the evaluation team were 
sensible but quite general. The leadership of the country was 
asked to make a clear political commitment to the promotion of 
literacy at the national level. The need for strengthening the 
system of advice through the activation of a national board for 
literacy was recommended as also the need to strengthen and 
support the system of administration and coordination from the 
center down to the village level. The team's report recommended 
that due attention be given to the establishment of linkages 
between the literacy agency and other development and extension 
agencies of the goverr~ent; and, finally, much greater emphasis 
on the training of personnel, at all the different levels of 
responsibility, in a variety of training settings such as courses, 
workshops, conferences and on-the-job training, was recommended. 
All these recommendations were offerings on the altar of technical 
rationality. 

The reconmendations on the administrative aspects of the 
program were much more important and concrete and served the 
arbitration needs of the project. The first recommendation 
related to the fact that the project which had been in o^ration 
for almost four years should have an exit date by which time 
there should be a transfer of responsibity from ON (A) and UN(B) 
to the agents of the Government of Xland (9). This exit date was 
set fcr December 31, 1985, sixteen months away from the date of the 
external evaluation. 

The chief of party, one of the two-man UN(B) project team, 
was asked to assume a new role as planning, programming and training 
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adviser instead of the current role which involved considerable 
amount of administrative duties including watchdog functions in 
regard to the properties contributed by UN (A) and UN(B) and 
super ivion of the work of IME. In other words , the role of the 
chief of party was changed from supervisory to advisory tr the 
government. The main purpose of this recommendation was, of 
course, to take responsibility away from the UN(B) J s official and 
give it to the government; and prepare the UN(B) project team for 
exit. It also freed the IME *rom the authority of the chief of 
party at least during the last few months of his stay at the 
project. The post of the chief of party was authorised until the 
end of the Litdev project. His particular role on the project 
and his demonstrated competence on the job stood him in good 
stead. 

The post oc the present IME was extended for four months 
only, exclusive of accrued leave. The government was requested 
to explore extra-budgetrary resources to fund the extension of 
the IME's post so that funds budgeted for consultant services are 
not used, if at all possible. In other words, the government was 
requested to let UN(B) off the hook for having requested and 
obtained funding in behalf of the government from a third p^rty, 
without prior consultation and without their formal request for 
such assistance. 

It should be clear from the nature and content of the 
recommendations made that the evaluation did indeed have an 
arbitration function. The way the recommendations were presented 
first in the "draft" form to a meeting of all stakeholders; and 
the counselling provided by the team leader to the IME who may 
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have felt to have lost the battle, were.* all in the nature of 
negotiating and healing. 

Lessons from the case 

The case presented above does lend support to the 
assertion made in the beginning of the paper that evaluation is a 
technology confounded by politics* Evaluat jns are indeed part 
of the prevailing politics as various stakeholders seek power, 
authority, advantage or accommodation within their social 
settings. 

While social evaluation is always political, it is by no 
means always for arbitration and negotiation. On the contrary, 
evaluation often can (and does) generate disagreement and feed 
conflict, as "facts" of the matter brought out or missed by 
an evaluation are used to support arguments and counterarguments. 
Thus, the use of evaluation for arbitration is just one of the 
many aspects of the politics of evaluation. We chose to bring 
tnis case to the present audience because we found the 
realization of evaluation as arbitration personally intriguing 
and theoretically interesting* The question now is: Was this a 
unique case, a universe of one, unlikely to be repeated 
elsewhere, or should we anticipate multiple instances of the use 
of eval* ^tion as arbitration in the world of education and 
development? 

In our view, the use of evaluation as arbitration is and 
will continue to be frequent. In a paper that I published many 
years ago, at the end of a two-year tour of duty with an UN- 
affiliated agency, I had suggested that the delivery of education 
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within the context of bilaterai ard multilateral technical 
assistance was indeed an act cf "educational diplomacy" (10). 
The diplomatic frame establishes the j. imitations as well as the 
possibilities of educational technical assistance. The 
limitations are real but do not render professional educational 
work impossible within the context of technical assistance. 
Indeed, an understanding of the limitations may make the 
actualization of possibilities more effective. 

The same seems to be true about educational and social 
evaluation. Tb^ conduct of evaluation within the context of 
bilateral or multilateral technical assistance is also an act of 
"evaluational diplomacy." The diplomatic frame establishes the 
limitations and possibilities of evaluation. Again, the 
limitations do not abhort the possibility of professsional 
evaluation. Indeed, the awareness of the political context may 
make the evaluation more useful than when no attention is p£.id to 
poltical realities and evaluators pretent to stand c^tside 
politics. Those of us who have worked with evaluations of 
educatic J and development programs in the technical assistance 
context in the Third World know, of course, that evaluations do 
evaluate processes and effects. Projects are reconceptualized or 
reorganized in the field, and some of them are terminated 
prematurely as a result of evaluations. 

In making the preceding statements, we are, of course, 
making epistemological claims in regard to the nature of facts in 
social life and the methods of generating knowledge for use in 
the imprr.dment of the human condition (11). We are assuming, on 
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the basis of fast accumulating experience, that in responding to 
the political realities of evaluation, the evaluator need not 
purposely engage in selective perception, or deliberately forget 
how to count, or ask trivial questions while real issues are 
avoided. Data need not be fabricated, nor is is necessary to 
offer misleading interpretations (12) • It is indeed possible to be 
both "scientific" and "moral" within the political context of our 
work. Evaluators can learn first to see and then to speak truth 
to power (13) • 

Toward a political theory of evaluation 

Current methodological debate in research and evaluation 
is coming to the view that a good theory of evaluation has to be 
a logico-dialectical theory. It must include the logic of 
systematic generation of information that is credible; and, at 
the same time, must be built upon the realization that both the 
processes of information generation and information utilization 
involve a complex of dialectical relationships between the 
evaluator, the evaluand, and all other stakeholders that have a 
stake in the evaluation and in the life of the program being 
evaluated. A useful theory of evaluation must enable the 
evaluator to "frame the flux" of the real world of social action. 
It must be able to accommodate both norms and counternorms, and both 
principled actions and selfish private needs. 

While the creative ideology of the new theory of 
evaluation must be logico-dialectical, the essential nature of the 
theory has to be political. Evaluation is political in its 
genesis, political as a process, and political in its effects. 
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Clearly, this means that evaluators must always begin with 
collecting political data of relevance to the evaluation action; 
and must learn to take account of political intelligence in the 
process of generating, interpreting and utilizing technical 
information. 

It is a challenge to the best of the evaluators in the 
business today to develop orientations, steps, and procedures 
that will ensure a symbiosis between the technology and politics 
of evaluation, ensuring that evaluation is of and above 
politics at the same time and that it contributes effectively to 
individual and social praxis. 
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